Interactive Analysis

• Background: Our Technique (why we have big data)
  • X-Ray Tomographic Microscopy
• Imaging in 2015
• The Problem(s)

The Tools

• Spark Imaging Layer
• 3D Imaging
• Hyperspectral Imaging
• Real-Time Streaming

The Science

• Genome Scale Studies
• Large Datasets
• Outlook / Developments

The only technique which can do all

• peer deep into large samples
• achieve \( \mathbf{<1\mu m} \) isotropic spatial resolution
  • with 1.8mm field of view
• achieve >10 Hz temporal resolution
• 8GB/s of images

[1] Mokso et al., J. Phys. D, 46(49),2013

Courtesy of M. Pistone at U. Bristol

X-Ray

• SRXTM (SLS) images at (>1000fps) \( \rightarrow \) 8GB/s
• cSAXS (SLS) thousands of diffraction patterns, acquisition rates might soon reach 30GB/s
• Nanoscopium (Soleil), 10TB/day, 10-500GB file sizes, very heterogenous data

Optical

• Light-sheet microscopy (see talk of Jeremy Freeman) produces images \( \rightarrow \) 500MB/s
• High-speed confocal images at (>200fps) \( \rightarrow \) 78Mb/s

Personal

• GoPro 4 Black - 60MB/s (3840 x 2160 x 30fps) for $600
• fps1000 - 400MB/s (640 x 480 x 840 fps) for $400

If you looked at one 1000 x 1000 sized image every second

it would take you
139 hours to browse through a terabyte of data.

Moores Law

\[ \textrm{Transistors} \propto 2^{T/(\textrm{18 months})} \]

Based on data from https://gist.github.com/humberto-ortiz/de4b3a621602b78bf90d

There are now many more transistors inside a single computer but the processing speed hasn't increased. How can this be?

• Multiple Core
  • Many machines have multiple cores for each processor which can perform tasks independently
• Multiple CPUs
  • More than one chip is commonly present
• New modalities
  • GPUs provide many cores which operate at slow speed

Parallel Code is important

The figure shows the range of cloud costs (determined by peak usage) compared to a local workstation with utilization shown as the average number of hours the computer is used each week.

The figure shows the cost of a cloud based solution as a percentage of the cost of buying a single machine. The values below 1 show the percentage as a number. The panels distinguish the average time to replacement for the machines in months

Here the equal cost point is shown where the cloud and local workstations have the same cost. The x-axis is the percentage of resources used at peak-time and the y shows the expected usable lifetime of the computer. The color indicates the utilization percentage and the text on the squares shows this as the numbers of hours used in a week.

X-Ray Microscopy isn't the first

Google, Facebook, Yahoo, and Amazon had these, or very similar problems years ago. They hired a lot of very competent Computer Science PhDs to solve it.

Cloud computing is ubiquitious and cheap

Cloud computing infrastructures are 10-100X cheaper than labor costs, usually more reliable internal systems, and easily accessible from almost anywhere.

The big why!

• You don't ask the questions you should, you ask the questions you can
• None of these approaches are robust or deal with the data flood
• 8 GB/s is more data than Facebook processes a day and 3 times as many images as Instagram (and they are all 5Mpx 14-bit and we don't compress them)

• http://news.cnet.com/8301-1023_3-57498531-93/facebook-processes-more-than-500-tb-of-data-daily/
• http://techcrunch.com/2013/01/17/instagram-reports-90m-monthly-active-users-40m-photos-per-day-and-8500-likes-per-second/

Correctness

The most important job for any piece of analysis is to be correct.

• A powerful testing framework is essential
• Avoid repetition of code which leads to inconsistencies
• Use compilers to find mistakes rather than users

Easily understood, changed, and used

Almost all image processing tasks require a number of people to evaluate and implement them and are almost always moving targets

• Flexible, modular structure that enables replacing specific pieces

Fast

The last of the major priorities is speed which covers both scalability, raw performance, and development time.

• Long waits for processing discourages exploration
• Manual access to data on separeate disks is a huge speed barrier
• Real-time image processing requires millisecond latencies
• Implementing new ideas can be done quickly

• Rather than building an analysis as quickly as possible and then trying to hack it to scale up to large datasets
  • chose the framework first
  • then start making the necessary tools.
• Google, Amazon, Yahoo, and many other companies have made huge in-roads into these problems
• The real need is a fast, flexible framework for robustly, scalably performing complicated analyses, a sort of Excel for big imaging data.

Apache Spark and Hadoop 2

The two frameworks provide a free out of the box solution for

• scaling to >10000 computers
• storing and processing exabytes of data
• fault tolerance
  • 2/3rds of computers can crash and a request still accurately finishes
• hardware and software platform indpendence (Mac, Windows, Linux)

These frameworks are really cool and Spark has a big vocabulary, but flatMap, filter, aggregate, join, groupBy, and fold still do not sound like anything I want to do to an image.

I want to

• filter out noise, segment, choose regions of interest
• contour, component label
• measure, count, and analyze
• …

Spark Image Layer

• Developed at 4Quant, ETH Zurich, and Paul Scherrer Institut
• The Spark Image Layer is a Domain Specific Language for Microscopy for Spark.
• It converts common imaging tasks into coarse-grained Spark operations

We have developed a number of commands for SIL handling standard image processing tasks

New components can be added using or imported from ImageJ directly. The resulting analyses are then parallelized by the Spark Engine and run on multiple cores/CPUs locally, on a cluster, supercomputer, or even a virtual in the cluster in the cloud which can be started in seconds.

Combining many different components together inside of the Spark Shell, IPython or Zeppelin, make it easier to assemble workflows

We want to understand the relationship between genetic background and bone structure

• With existing tools, analysis is possible and a number of publications have been made, even ones that show differences between strains of mice
• But
  • n<12
  • time-consuming (years between measurement and publication)
  • not flexible or reproducible
  • not cloud-based

Genetic studies require hundreds to thousands of samples, in this case the difference between 717 and 1200 samples is the difference between finding the links and finding nothing.

2008 approach - 120 years

• Hand Identification -> 30s / object
• 30-40k objects per sample
• One Sample in 6.25 weeks

2014 approach - 1.5 years

• ImageJ macro for segmentation (2-4 hours / sample)
• Python script for shape analysis (3 hours / sample)
• Paraview macro for network and connectivity (2 hours / sample)
• Python script to pool results (3-4 hours)
• MySQL Database storing results (5 minutes / query)

• Collaboration with A. Astolfo and A. Patera
• Measure a full mouse brain (1cm\( ^3 \)) with cellular resolution (1 \( \mu \) m)
• 10 x 10 x 10 scans at 2560 x 2560 x 2160 = 14TVoxels

• 14TVoxels = 56TB
• Each scan needs to be registered and aligned together
• There are no computers with 56TB of memory
• Even multithreaded approachs are not feasible and require many logistics
• Analysis of the stitched data is also of interest (segmentation, vessel analysis, distribution and network connectivity)

\[ \textrm{Images}: \textrm{RDD}[((x,y,z),Img[Double])] =\\ \left[(\vec{x},\textrm{Img}),\cdots\right] \]

dispField = Images.
  cartesian(Images).map{
  case ((xA,ImA), (xB,ImB)) =>
    xcorr(ImA,ImB,in=xB-xA)
  }

From the updated information provided by the cross correlations and by applying appropriate smoothing criteria (if necessary).

The stitching itself, rather than rewriting the original data can be done in a lazy fashion as certain regions of the image are read.

def getView(tPos,tSize) =
  stImgs.
  filter(x=>abs(x-tPos)<img.size).
  map { case (x,img) =>
   val oImg = new Image(tSize)
   oImg.copy(img,x,tPos)
}.addImages(AVG)

This also ensures the original data is left unaltered and all analysis is reversible.

In the biological imaging community, the open source tools of ImageJ2 and Fiji are widely accepted and have a large number of readily available plugins and tools.

We can integrate the functionality directly into Spark and perform operations on much larger datasets than a single machine could have in memory. Additionally these analyses can be performed on streaming data.

Apache Spark is brilliant platform and utilizing GraphX, MLLib, and other packages there unlimited possibilities

• Scala is a beautiful but not easy language
• Python is an easier language
• Both suffer from
  • Namespaces
  • Non obvious workflows

• Although all analyses can be expressed as a workflow, this is often difficult to see from the code
• Non-technical persons have little ability to understand or make minor adjustments to analysis
• Parameters require recompiling to change
• or GUIs need to be placed on top

Here we use a KNIME-based workflow and our Spark Imaging Layer extensions to create a workflow without any Scala or programming knowledge and with an easily visible flow from one block to the next without any performance overhead of using other tools.

A spinoff - 4Quant: From images to insight

• Cloud Image Processing
  • Use our distributed version of ImageJ in the cloud to analyze thousands of remote datasets using your own, ours, or community provided processing routines
• Custom Analysis Solutions
  • Custom-tailored software to solve your problems
• One Stop Shop
  • Measurement, analysis, and statistical analysis

Education / Training

• Consulting
  • Advice on imaging techniques, analysis possibilities
  • Development of new analysis tools and workflows
• Education
  • Workshops on Image Analysis
  • Courses / Training
  • Quantitative Big Imaging

• Spark is not performant \( \rightarrow \) dedicated, optimized CPU and GPU codes will perform slightly to much much better when evaulated by pixels per second per processing power unit
  • these codes will be wildly outperformed by dedicated hardware / FPGA solutions
• Serialization overhead and network congestion are not neglible for large datasets

But

• Scala / Python in Spark is substantially easier to write and test
  • Highly optimized codes are very inflexible
  • Human time is 400x more expensive than AWS time
  • Mistakes due to poor testing can be fatal
• Spark scales smoothly to enormous datasets
  • GPUs rarely have more than a few gigabytes
  • Writing code that pages to disk is painful
• Spark is hardware agnostic (no drivers or vendor lock-in)

For many datasets processing, segmentation, and morphological analysis is all the information needed to be extracted. For many systems like bone tissue, cellular tissues, cellular materials and many others, the structure is just the beginning and the most interesting results come from the application to physical, chemical, or biological rules inside of these structures.

\[ \sum_j \vec{F}_{ij} = m\ddot{x}_i \]

Such systems can be easily represented by a graph, and analyzed using GraphX in a distributed, fault tolerant manner.

Bottleneck is filesystem connection, many nodes (10+) reading in parallel brings even GPFS-based infiniband system to a crawl

One of the central tenants of MapReduce™ is data-centric computation \( \rightarrow \) instead of data to computation, move the computation to the data.

• Use fast local storage for storing everything redundantly \( \rightarrow \) less transfer and fault-tolerance
• Largest file size: 512 yottabytes, Yahoo has 14 petabyte filesystem in use